{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some features to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run the following from the cs231n directory and try again:\n",
      "python setup.py build_ext --inplace\n",
      "You may also need to restart your iPython kernel\n"
     ]
    }
   ],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.3\n",
      "Mean of input:  10.0002078785\n",
      "Mean of train-time output:  9.98526577052\n",
      "Mean of test-time output:  10.0002078785\n",
      "Fraction of train-time output set to zero:  0.300876\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.6\n",
      "Mean of input:  10.0002078785\n",
      "Mean of train-time output:  10.035153558\n",
      "Mean of test-time output:  10.0002078785\n",
      "Fraction of train-time output set to zero:  0.598632\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.75\n",
      "Mean of input:  10.0002078785\n",
      "Mean of train-time output:  10.0216257491\n",
      "Mean of test-time output:  10.0002078785\n",
      "Fraction of train-time output set to zero:  0.749504\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.3, 0.6, 0.75]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  1.89290486527e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.8, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specificially, if the constructor the the net receives a nonzero value for the `dropout` parameter, then the net should add dropout immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  0\n",
      "Initial loss:  2.30047908977\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.25\n",
      "Initial loss:  2.30486921512\n",
      "W1 relative error: 2.47e-07\n",
      "W2 relative error: 1.69e-06\n",
      "W3 relative error: 8.70e-07\n",
      "b1 relative error: 7.84e-09\n",
      "b2 relative error: 6.85e-09\n",
      "b3 relative error: 1.13e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.31013690872\n",
      "W1 relative error: 2.57e-08\n",
      "W2 relative error: 1.49e-08\n",
      "W3 relative error: 4.49e-08\n",
      "b1 relative error: 1.35e-09\n",
      "b2 relative error: 1.91e-09\n",
      "b3 relative error: 9.51e-11\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [0, 0.25, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a dropout probability of 0.75. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "(Iteration 1 / 125) loss: 7.856644\n",
      "(Epoch 0 / 25) train acc: 0.274000; val_acc: 0.192000\n",
      "(Epoch 1 / 25) train acc: 0.410000; val_acc: 0.263000\n",
      "(Epoch 2 / 25) train acc: 0.518000; val_acc: 0.269000\n",
      "(Epoch 3 / 25) train acc: 0.550000; val_acc: 0.248000\n",
      "(Epoch 4 / 25) train acc: 0.684000; val_acc: 0.297000\n",
      "(Epoch 5 / 25) train acc: 0.758000; val_acc: 0.292000\n",
      "(Epoch 6 / 25) train acc: 0.782000; val_acc: 0.266000\n",
      "(Epoch 7 / 25) train acc: 0.860000; val_acc: 0.240000\n",
      "(Epoch 8 / 25) train acc: 0.864000; val_acc: 0.285000\n",
      "(Epoch 9 / 25) train acc: 0.898000; val_acc: 0.279000\n",
      "(Epoch 10 / 25) train acc: 0.910000; val_acc: 0.269000\n",
      "(Epoch 11 / 25) train acc: 0.948000; val_acc: 0.292000\n",
      "(Epoch 12 / 25) train acc: 0.960000; val_acc: 0.287000\n",
      "(Epoch 13 / 25) train acc: 0.952000; val_acc: 0.282000\n",
      "(Epoch 14 / 25) train acc: 0.952000; val_acc: 0.267000\n",
      "(Epoch 15 / 25) train acc: 0.944000; val_acc: 0.287000\n",
      "(Epoch 16 / 25) train acc: 0.936000; val_acc: 0.265000\n",
      "(Epoch 17 / 25) train acc: 0.950000; val_acc: 0.278000\n",
      "(Epoch 18 / 25) train acc: 0.968000; val_acc: 0.294000\n",
      "(Epoch 19 / 25) train acc: 0.970000; val_acc: 0.286000\n",
      "(Epoch 20 / 25) train acc: 0.980000; val_acc: 0.304000\n",
      "(Iteration 101 / 125) loss: 0.204318\n",
      "(Epoch 21 / 25) train acc: 0.952000; val_acc: 0.289000\n",
      "(Epoch 22 / 25) train acc: 0.982000; val_acc: 0.289000\n",
      "(Epoch 23 / 25) train acc: 0.970000; val_acc: 0.284000\n",
      "(Epoch 24 / 25) train acc: 0.986000; val_acc: 0.303000\n",
      "(Epoch 25 / 25) train acc: 0.984000; val_acc: 0.284000\n",
      "0.75\n",
      "(Iteration 1 / 125) loss: 19.352448\n",
      "(Epoch 0 / 25) train acc: 0.244000; val_acc: 0.199000\n",
      "(Epoch 1 / 25) train acc: 0.414000; val_acc: 0.256000\n",
      "(Epoch 2 / 25) train acc: 0.472000; val_acc: 0.288000\n",
      "(Epoch 3 / 25) train acc: 0.502000; val_acc: 0.272000\n",
      "(Epoch 4 / 25) train acc: 0.562000; val_acc: 0.287000\n",
      "(Epoch 5 / 25) train acc: 0.628000; val_acc: 0.293000\n",
      "(Epoch 6 / 25) train acc: 0.612000; val_acc: 0.289000\n",
      "(Epoch 7 / 25) train acc: 0.658000; val_acc: 0.285000\n",
      "(Epoch 8 / 25) train acc: 0.706000; val_acc: 0.289000\n",
      "(Epoch 9 / 25) train acc: 0.706000; val_acc: 0.294000\n",
      "(Epoch 10 / 25) train acc: 0.758000; val_acc: 0.304000\n",
      "(Epoch 11 / 25) train acc: 0.736000; val_acc: 0.307000\n",
      "(Epoch 12 / 25) train acc: 0.778000; val_acc: 0.303000\n",
      "(Epoch 13 / 25) train acc: 0.782000; val_acc: 0.306000\n",
      "(Epoch 14 / 25) train acc: 0.796000; val_acc: 0.324000\n",
      "(Epoch 15 / 25) train acc: 0.792000; val_acc: 0.336000\n",
      "(Epoch 16 / 25) train acc: 0.828000; val_acc: 0.318000\n",
      "(Epoch 17 / 25) train acc: 0.876000; val_acc: 0.313000\n",
      "(Epoch 18 / 25) train acc: 0.826000; val_acc: 0.320000\n",
      "(Epoch 19 / 25) train acc: 0.870000; val_acc: 0.293000\n",
      "(Epoch 20 / 25) train acc: 0.880000; val_acc: 0.298000\n",
      "(Iteration 101 / 125) loss: 3.992197\n",
      "(Epoch 21 / 25) train acc: 0.878000; val_acc: 0.309000\n",
      "(Epoch 22 / 25) train acc: 0.888000; val_acc: 0.320000\n",
      "(Epoch 23 / 25) train acc: 0.904000; val_acc: 0.313000\n",
      "(Epoch 24 / 25) train acc: 0.910000; val_acc: 0.298000\n",
      "(Epoch 25 / 25) train acc: 0.928000; val_acc: 0.310000\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [0, 0.75]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4IAAAJNCAYAAABkwMu0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X+cXGV58P/PZQiyBcyKUDWb0ATBCCYx0S22jVYfsQZU\nIKBGsIpoLQV/QH2epoW+WsoX25KaVpRHlCdSrPiDmFaIUMHYBhXBX2xMmvDDSEQ0u6AGMJEfiyTh\n+v4xs2F2s0l2MztzduZ83q/XvmbOfc6cc83MybIX931fd2QmkiRJkqTyeEbRAUiSJEmSmstEUJIk\nSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSVBoRMSEiHo2Iw4uORZKkIoXr\nCEqSxquIeLRm87eA3wA7qtt/lpmfb35UkiS1PhNBSVJLiIj7gPdk5n/v4Zj9MnN786JqnnZ+b5Kk\n5nNoqCSpZUXE30fEFyPimoh4BHh7RPx+RHw3IrZExAMRcVlETKwev19EZERMq25/rrr/poh4JCK+\nExHTd3OtZ0TEf0TEz6vn/kZEHF2z/7ci4tKI+FlEbI2IWyLimdV9f1iNaWtEbIqId1Tbb42IM2vO\n8Z6I+MaQWN8bERuBH1bbPx4RvRHx64i4PSL+oOb1+0XE30bEj6v7eyJickT8v4j4pyHv58aI+ED9\n34IkqRWZCEqSWt0pwBeAScAXge3AecChwDzgeODP9vD6twF/CxwC/Az40B6O/U/gKOB5wB3AZ2v2\nXQrMBl5ePddfA09VE8sbgY8AzwHmAutH8f5OAn4XmFXd/l71OocA/wH8+0DCCSwC3kzlPXcC7wGe\nAD4DnB4RARARzwVeDVwzijgkSW3ERFCS1OpuzcwbMvOpzOzPzNsz83uZuT0z7wWWAq/aw+v/IzN7\nMnMb8HlgznAHVc//b5n5SGY+AVwEvCwiDoyICcCZwLmZ+UBm7sjMW6vnfDtwU2Yur8b0YGauHcX7\n+8fM/FVm9lfj+GxmPlwdJvph4FnAkdVj3wP8dWbeU413bfXYb1NJCF9dPe504L8z88FRxCFJaiMm\ngpKkVrepdiMiXhQRX6kO4fw1cDGV3sHd+XnN88eBg4Y7qFpx9MMRcW/1vBuruw4FngvsD/x4mJdO\n3U37SA19f38ZET+MiK3Ar4ADefr97elaV1NJSqk+fnY3x0mSSsBEUJLU6oZWPft/VIZtHpmZzwIu\nBGIMrnMG8HrgNVSGoQ70wgXwC+BJ4AXDvG7TbtoBHqNSDXXA84Y5Zuf7i4j/Bfxv4E1Uhn4+G3iU\np9/fnq71WeCUiJhbPeaG3RwnSSoBE0FJUrs5GNgKPFYt5rKn+YGjPe9vgIeoJG//MLAjM3cA/wZ8\nNCKeV+09nFctUvM54PiIeFO1mMuhEfGS6kvXAm+KiI6IeCHw7hHEsB14EJhIZXjqgTX7rwT+PiJe\nEBVzIuKQaow/Bf6HynzBf68Ob5UklZSJoCSp3fwf4J3AI1R6B784Ruf9NHB/9edO4NtD9n8QuBtY\nDTwM/COVZZp+ApwI/FW1/Qc8Xfjln6n0+P0SuIpK0rgnNwL/DdwD3Af8GnigZv8SYAWwqrpvKXBA\nzf7PVK/tsFBJKjnXEZQkqSQi4jXAvwJHpH8ASFKp2SMoSVIJRMT+VJbV+JRJoCTJRFCSpDYXEbOo\nVBg9BLis4HAkSeOAQ0MlSZIkqWTsEZQkSZKkkjERlCRJkqSS2a/oAMbSoYcemtOmTSs6DEmSJEkq\nxOrVqx/MzMP2dlxbJYLTpk2jp6en6DAkSZIkqRAR8dORHOfQUEmSJEkqmUISwYi4KiJ+GRF37GZ/\nRMRlEbExItZFxEubHaMkSZIktauiegT/DTh+D/tPAI6q/pwFfLIJMUmSJElSKRSSCGbmLcDDezjk\nZODqrPgu0BkRz29OdJIkSZLU3sbrHMEuYFPNdm+1TZIkSZJUp/GaCI5YRJwVET0R0bN58+aiw5Ek\nSZKkcW+8Lh/RB0yt2Z5SbdtFZi4FlgJ0d3dn40OTJEmSVKQVa/pYsnID92/pZ3JnB4vmz2DBXAcQ\njsZ4TQSvB94fEcuAlwNbM/OBgmOSJEmSxp2yJUUr1vRxwbXr6d+2A4C+Lf1ccO16gKa873b5vAtJ\nBCPiGuDVwKER0Qv8HTARIDOvAG4EXg9sBB4H3lVEnJIkSWot7fJH+kgVnRQVYcnKDTvf74D+bTtY\nsnJDw99zO33ehSSCmXn6XvYn8L4mhSNJkqQ20E5/pI9UkUlRUe7f0j+q9rHUTp93yxeLkSRJkmDP\nf6S3qyKToqJM7uwYVftYaqfPe7zOEZQkSVKdihwmWcS12+mP9JGa3NlB3zDvrxlJERTzPS+aP2NQ\nzy9Ax8QJLJo/o6HXheI/77Fkj6AkSVIbGhgm2beln+TpYZIr1gxbiL0trl1kT1FRFs2fQcfECYPa\nmpUUFfU9L5jbxSWnzqKrs4MAujo7uOTUWU35nxxFft5jzR5BSZKkNlTkXKairl1kT1FRBj7PInp+\ni7zHFsztKmROXpGf91gzEZQkifJVGlT7K3KYZFHXbqc/0kejqKSojENxobjPe6yZCEqSSq+MlQbV\n/oqcy1Tktdvlj/RW0E7z5crIOYKSpNIrstLgijV9zFt8M9PP/wrzFt/clPlb4+Haarwi5zK10zyq\n0Sjbv6myfs/twh5BSdK4UdTwzKKGNxXZE2kvaHMVcW8XOUyyjEM0y/hvqozfczuJytrt7aG7uzt7\nenqKDkOStA+G/hEFlf+z3IxKcPMW3zzs8Kauzg5uO/81bXfdoq9dtvmYRd7bap4i/01JtSJidWZ2\n7+04h4ZKksaFIodnFjW8qYzFPIpc0qAoZVzkvIzKWjhFrctEUJI0LhT5R1RRa1IVueZZUdcuY1Jk\nglAOZVzDUK3NOYKSpHGh6OpzRVQaLHLNs6KuXXRSVMSw1KLvbTVHGdcwVGuzR1CSNC6UsfpcUT2R\nRV67yF6TooallvHeLqMi/z1L+8JiMZKkcaNsRUTKqIxFgcB7W2or65bDqothay9MmgLHXQizFxYd\n1U4jLRbj0FBJ0rjhQtDtr8hy80XPQ/XeltrAuuVww7mwrfp7Y+umyjaMq2RwJEwEJUlSUxWVFDlX\nT1LdVl38dBI4YFt/pb3FEkHnCEqSpFJwrp6kum3tHV37OGaPoCRJKoUih6VKahOTplSGgw7X3mJM\nBCVJu7CwhdqVc/WkNlFUwZbjLhw8RxBgYkelvcWYCEqSBhla1XGgxD7gH9CSpOIVWbBl4PzjuGro\nSBWyfEREHA98DJgAXJmZi4fsfzZwFfAC4Ang3Zl5x97O6/IRklS/IkvsS5JaTBE9c5fO3M3wzKnw\nwb2mDG1vpMtHNL1YTERMAC4HTgCOAU6PiGOGHPbXwNrMnA2cQSVplCQ1QZEl9iVJLWSgZ27rJiCf\n7plbt7yx122jgi1FKqJq6LHAxsy8NzOfBJYBJw855hjgZoDM/CEwLSKe29wwJal4K9b0MW/xzUw/\n/yvMW3wzK9b0Nfyauyulb4l9SdIge1pKoZF2V5ilBQu2FKmIRLALqO3L7a221fof4FSAiDgW+B3A\nb1ZSqQzM1evb0k/y9Fy9RieDltiXJI1IUT1zx11YKdBSq0ULthRpvK4juBjojIi1wAeANcCO4Q6M\niLMioiciejZv3tzMGCWpoZas3LCzYMuA/m07WLJyQ0Ovu2BuF5ecOouuzg6CytzAS06dZaEYSdJg\nRfXMzV4IJ15WmRNIVB5PvKwlC7YUqYiqoX3A1JrtKdW2nTLz18C7ACIigJ8A9w53ssxcCiyFSrGY\nBsQrSYUocq6eJfYlqYWUcSmF2QtN/OpURI/g7cBRETE9IvYHTgOurz0gIjqr+wDeA9xSTQ4lqTSc\nqydJ2quiCraAPXMtruk9gpm5PSLeD6yksnzEVZl5Z0ScXd1/BXA08JmISOBO4E+aHackFW3R/BmD\n1vMD5+pJ0l4V1TtWlD0VbGnG+7ZnrmUVsqB8Zt4I3Dik7Yqa598BXtjsuCRpPBkYmrlk5Qbu39LP\n5M4OFs2f4ZBNqRWVLTkpSpELjQ9cv9nfs0spaB8VkghKkkbGuXpSGyg6OSmTInvHivqeJ03ZzeLq\nFtzXno3XqqGSJEntoai11oq2bjlcOhMu6qw8NmPOWpG9Y0V9zy6loH1kIihJktRIZRy6V1QBkyIX\nGi/qe7Zgi/aRQ0MlaS9WrOlznp6kfVfGoXtFDdEscjmDIr9nC7ZoH9gjKKllrFjTx7zFNzP9/K8w\nb/HNrFjTt/cXjcE1L7h2PX1b+kmgb0s/F1y7vinXltQmyjh0r4y9Y2X8ntXS7BGU1BIGErKBpRQG\nEjKgob1zS1ZuGLR8A0D/th0sWbnBXkFJIzOQhJSpamgZe8fK+D2rpZkISmoJRSVk92/pH1W7JA2r\nbEP3ihyiWaSyfc9qaQ4NldQSikrIJnd2jKpdkoQFTKQWYI+gpJYwubODvmGSvkYnZIvmzxg0JBWg\nY+IEFs2f0dDrSlLLs3dMGtfsEZTUEhbNn0HHxAmD2pqRkC2Y28Ulp86iq7ODALo6O7jk1FnOD5Qk\nSS3NHkFJLWEg8SpiGYcFc7tM/CRJUlsxEZTUMkzIJEmSxoZDQyVJkiSpZEwEJUmSJKlkTAQlSVJ5\nrFsOl86Eizorj+uWFx1RY5Xt/UoaMecISpKkcli3fPAi51s3VbahPZc5KNv7lTQq9ghKkqRyWHXx\n00nRgG39lfZ2VLb3K2lUTAQlSVI5bO0dXXurK9v7lTQqJoKSJKkcJk0ZXXurK9v7lTQqJoKSJKkc\njrsQJnYMbpvYUWlvR2V7v5JGxURQkiSVw+yFcOJlMGkqEJXHEy9r38IpZXu/kkYlMrP5F404HvgY\nMAG4MjMXD9k/CfgccDiVyqb/nJmf3tt5u7u7s6enpwERS5IkSdL4FxGrM7N7b8c1vUcwIiYAlwMn\nAMcAp0fEMUMOex9wV2a+BHg18C8RsX9TA5UkSZKkNlXE0NBjgY2ZeW9mPgksA04eckwCB0dEAAcB\nDwPbmxumJEltzsXGJam0ilhQvgvYVLPdC7x8yDEfB64H7gcOBt6amU81JzxJkkqgyMXG1y2vrGW3\ntbdSwfK4C523JklNNl6LxcwH1gKTgTnAxyPiWcMdGBFnRURPRPRs3ry5mTFKktS6ilpsfCAB3boJ\nyKcTUHsjJampikgE+4CpNdtTqm213gVcmxUbgZ8ALxruZJm5NDO7M7P7sMMOa0jAkiS1naIWGy8q\nAZUkDVLE0NDbgaMiYjqVBPA04G1DjvkZcBzwrYh4LjADuLepUUoa1oo1fSxZuYH7t/QzubODRfNn\nsGBuV9FhSa2rqGGSk6ZUe+WGaW+kohJQSdIgTe8RzMztwPuBlcDdwPLMvDMizo6Is6uHfQj4g4hY\nD6wC/iozH2x2rJIGW7GmjwuuXU/fln4S6NvSzwXXrmfFmqGd+lILKqJwSpHDJItabHx3iWajE1BJ\n0iCFrCPYKK4jKDXWvMU307elf5f2rs4Objv/NQVEJI2RoYVToJIUNXrx7Utn7qZXbip88I7GXXdA\nEb2RRX3WklQSI11HsIihoZJa1P3DJIF7apdaxp7mrTUyOSl6mOTshc1PvgauZ9VQSSqUiaCkEZvc\n2TFsj+Dkzo5hjpZaSFEJWVHz9IpWRAIqSRpkvC4fIWkcWjR/Bh0TJwxq65g4gUXzZxQUkTRGipq3\nVtQ8PUlS6ZkIShqxBXO7uOTUWXR1dhBU5gZecuosq4aq9RWVkM1eWJkbN2kqEJVH58pJkprAYjFS\ni3IZBzVUUUsaFKmM71mS1HYsFiO1sYFlHPq37QCeXsYBMBlU/YZWdRxY0gDaOzFy3pokqUQcGiq1\noCUrN+xMAgf0b9vBkpUbCopIbWVPFTQlSVJbMBGUWpDLOKihil7SQJIkNZyJoNSCdrdcg8s4aEwU\nVUFTkiQ1jYmg1IJcxkEN5ZIGkiS1PYvFSC1ooCCMVUPVEAMFU6ygKUlS23L5CEnS+OESDpIk1cXl\nIyRJraWsy1ZIklQA5whKksYHl62QJKlpTAQlSeODy1ZIktQ0JoKSpPHBZSskSWoaE0FJ0vjgshWS\nJDWNiaAkaXyYvRBOvAwmTQWi8njiZRaKkSSpAawaKkkaP2YvNPGTJKkJ7BGUpL1ZtxwunQkXdVYe\n1y0vx7UlSVLbKiQRjIjjI2JDRGyMiPOH2b8oItZWf+6IiB0RcUgRsUoquYG17bZuAvLpte2akZAV\neW1JktTWmp4IRsQE4HLgBOAY4PSIOKb2mMxckplzMnMOcAHwzcx8uNmxSlKha9u5rp4kSWqQInoE\njwU2Zua9mfkksAw4eQ/Hnw5c05TIJGmoIte2c109SZLUIEUkgl3Apprt3mrbLiLit4DjgS81IS5J\n2lWRa9u5rp4kSWqQ8V4s5kTgtj0NC42IsyKiJyJ6Nm/e3MTQJJVCkWvbua6eJElqkCISwT5gas32\nlGrbcE5jL8NCM3NpZnZnZvdhhx02RiFKUlWRa9u5rp4kSWqQyMzmXjBiP+BHwHFUEsDbgbdl5p1D\njpsE/ASYmpmPjeTc3d3d2dPTM8YRS7u3Yk0fS1Zu4P4t/Uzu7GDR/BksmDvsSGdJkiSp4SJidWZ2\n7+24pi8on5nbI+L9wEpgAnBVZt4ZEWdX919RPfQU4GsjTQKlZluxpo8Lrl1P/7YdAPRt6eeCa9cD\nmAxKkiRpXKurRzAiPgB8LjN/NXYh7Tt7BNVM8xbfTN+W/l3auzo7uO381xQQkSRJkspupD2C9c4R\nfC5we0Qsry4SH3WeT2oZ9w+TBO6pXZIkSRov6koEM/NvgKOAfwXOBO6JiH+MiBeMQWzSuDa5s2NU\n7ZIkSdJ4UXfV0KyMLf159Wc78GzgPyLiw/WeWxrPFs2fQcfECYPaOiZOYNH8GQVFJEmSJI1MXcVi\nIuI84AzgQeBKYFFmbouIZwD3AH9Zf4jS+DRQEMaqoZIkSWo19VYNPQQ4NTN/WtuYmU9FxBvrPLc0\n7i2Y22Xi10zrlsOqi2FrL0yaUllY3TX1JEmSRq3eoaE3AQ8PbETEsyLi5QCZeXed55akp61bDjec\nC1s3AVl5vOHcSrskSZJGpd5E8JPAozXbj1bbJGlsrboYtg2pyLqtv9IuSZKkUak3EYysWYgwM5+i\ngEXqJZXA1t7RtUuSJGm36k0E742IcyNiYvXnPODesQhMkgaZNGV07ZIkSdqtehPBs4E/APqAXuDl\nwFn1BiVJuzjuQpg4ZI3GiR2VdkmSJI1KXcM4M/OXwGljFIsk7d5AdVCrhkqSJNWt3nUEDwD+BHgx\ncMBAe2a+u864JGlXsxea+EmSJI2BeoeGfhZ4HjAf+CYwBXik3qAkSZIkSY1TbyJ4ZGb+LfBYZn4G\neAOVeYKSGm3dcrh0JlzUWXl0PT1JkiSNUL1LPWyrPm6JiJnAz4HfrvOckvZmYHH1gXX1BhZXB4dO\nSpIkaa/q7RFcGhHPBv4GuB64C/inuqOStGcuri5JkqQ67HOPYEQ8A/h1Zv4KuAU4YsyikrRnLq4u\nSZKkOuxzj2BmPgX85RjGImmkXFxdkiRJdah3aOh/R8RfRMTUiDhk4GdMIpO0e0Uurm6RGkmSpJZX\nb7GYt1Yf31fTljhMVGqsohZXt0iNJElSW4jMLDqGMdPd3Z09PT1FhyG1r0tnVpK/oSZNhQ/e0fx4\nJEmSNEhErM7M7r0dV1ePYEScMVx7Zl69l9cdD3wMmABcmZmLhznm1cBHgYnAg5n5qnpilTQGLFIj\nSZLUFuodGvq7Nc8PAI4DfgDsNhGMiAnA5cAfAb3A7RFxfWbeVXNMJ/AJ4PjM/FlEuDahNB5MmrKb\nHkGL1EiSJLWSuhLBzPxA7XY1gVu2l5cdC2zMzHurr1kGnExlDcIBbwOuzcyfVa/zy3rilDRGjrtw\n8BxBaF6RGkmSJI2ZequGDvUYMH0vx3QBtV0KvdW2Wi8Enh0R34iI1bsbgiqpyWYvhBMvq8wJJCqP\nJ15moRhJkqQWU+8cwRuoVAmFSlJ5DDAWteT3A15GZahpB/CdiPhuZv5omBjOAs4COPzww8fg0pL2\naPZCEz9JkqQWV+8cwX+ueb4d+Glm7q1qRB8wtWZ7SrWtVi/wUGY+BjwWEbcALwF2SQQzcymwFCpV\nQ0cXviRJkiSVT71DQ38GfC8zv5mZtwEPRcS0vbzmduCoiJgeEfsDpwHXDznmy8ArImK/iPgt4OXA\n3XXGKkmSJEmi/kTw34GnarZ3VNt2KzO3A+8HVlJJ7pZn5p0RcXZEnF095m7gq8A64PtUlphwkTJJ\nkiRJGgP1Dg3dLzOfHNjIzCervXx7lJk3AjcOabtiyPYSYEmd8UmSJEmShqi3R3BzRJw0sBERJwMP\n1nlOSZIkSVID1dsjeDbw+Yj4eHW7F3CpB0mSJEkax+pdUP7HwO9FxEHV7UfHJCpJkiRJUsPUNTQ0\nIv4xIjoz89HMfDQinh0Rfz9WwUnj3rrlcOlMuKiz8rhuLJbRlCRJkhqr3jmCJ2TmloGNzPwV8Po6\nzym1hnXL4YZzYesmICuPN5xrMihJkqRxr95EcEJEPHNgIyI6gGfu4Xipfay6GLb1D27b1l9plyRJ\nksaxeovFfB5YFRGfBgI4E/hMvUFJo7ViTR9LVm7g/i39TO7sYNH8GSyY29XYi27tHV27JEmSNE7U\nWyzmnyLif4DXAkllkfjfGYvApJFasaaPC65dT/+2HQD0benngmvXAzQ2GZw0pTosdJh2SZIkaRyr\nd2gowC+oJIFvAV4D3D0G55RGbMnKDTuTwAH923awZOWGxl74uAthYsfgtokdlXZJkiRpHNunHsGI\neCFwevXnQeCLQGTm/xrD2KQRuX9L/6jax8zshZXHVRdXhoNOmlJJAgfaJUmSpHFqX4eG/hD4FvDG\nzNwIEBEfHLOopFGY3NlB3zBJ3+TOjmGOHmOzF5r4SZIkqeXs69DQU4EHgK9HxKci4jgqxWKkpls0\nfwYdEycMauuYOIFF82cUFJEkSZI0vu1TIpiZKzLzNOBFwNeBPwd+OyI+GRGvG8sApb1ZMLeLS06d\nRVdnBwF0dXZwyamzGl81VJIkSWpRkZljc6KIZ1MpGPPWzDxuTE46St3d3dnT01PEpSVJkiSpcBGx\nOjO793bcWFQNBSAzf5WZS4tKAiVJkiRJIzNmiaAkSZIkqTXUtaC8VGvFmj6WrNzA/Vv6mdzZwaL5\nM5ynJ0mSJI1DJoIaEyvW9HHBtet3Luzet6WfC65dD2AyKEmSJI0zDg3VmFiycsPOJHBA/7YdLFm5\noaCIJEmSJO2OiaDGxP3DLOi+p3ZJkiRJxSkkEYyI4yNiQ0RsjIjzh9n/6ojYGhFrqz8XFhGnRm5y\nZ8eo2sfcuuVw6Uy4qLPyuG55c64rSZIktaCmJ4IRMQG4HDgBOAY4PSKOGebQb2XmnOrPxU0NUqO2\naP4MOiZOGNTWMXECi+bPaPzF1y2HG86FrZuArDzecK7JoCRJkrQbRfQIHgtszMx7M/NJYBlwcgFx\naAwtmNvFJafOoquzgwC6Oju45NRZzSkUs+pi2DZkCOq2/kq7JEmSpF0UUTW0C9hUs90LvHyY4/4g\nItYBfcBfZOadzQhO+27B3K5iKoRu7R1duyRJklRy47VYzA+AwzNzNvB/gRW7OzAizoqInojo2bx5\nc9MC1Dgyacro2iVJkqSSKyIR7AOm1mxPqbbtlJm/zsxHq89vBCZGxKHDnSwzl2Zmd2Z2H3bYYY2K\nWePZcRfCxCFFaSZ2VNolSZIk7aKIRPB24KiImB4R+wOnAdfXHhARz4uIqD4/lkqcDzU9UrWG2Qvh\nxMtg0lQgKo8nXlZplyRJkrSLps8RzMztEfF+YCUwAbgqM++MiLOr+68A3gycExHbgX7gtMzMZseq\nFjJ7oYmfJEmSNELRTvlVd3d39vT0FB2GJEmSJBUiIlZnZvfejhuvxWLUilzUXZIkSWoJRSwfoXY0\nsKj7wHp+A4u6g0M2JUmSpHHGHkGNDRd1lyRJklqGiaDGhou6S5IkSS3DRFBjw0XdJUmSpJZhIqix\n4aLukiRJUsswEdTYcFF3SZIkqWVYNVRjx0XdJUmSpJZgj6AkSZIklYyJoCRJkiSVjImgJEmSJJWM\niaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiWA7WrccLp0J\nF3VWHtctLzoiSZIkSePIfkUHoDG2bjnbv/wB9tvxRGV766bKNsDshUVGJkmSJGmcsEewzTx+04VP\nJ4FV++14gsdvurCgiCRJkiSNN4UkghFxfERsiIiNEXH+Ho773YjYHhFvbmZ8reyA/p+Pql2SJElS\n+TQ9EYyICcDlwAnAMcDpEXHMbo77J+BrzY2wtd3/1HNG1S5JkiSpfIroETwW2JiZ92bmk8Ay4ORh\njvsA8CXgl80MrtVduf/beTz3H9T2eO7Plfu/vaCIJEmSJI03RSSCXcCmmu3eattOEdEFnAJ8solx\ntYU5bziLC/Msep86lKcy6H3qUC7Ms5jzhrOKDk2SJEnSODFeq4Z+FPirzHwqIvZ4YEScBZwFcPjh\nhzchtPFtwdwu4L28deVx3L+ln8mdHSyaP6PaLkmSJEnFJIJ9wNSa7SnVtlrdwLJqEngo8PqI2J6Z\nK4aeLDOXAksBuru7syERt5gFc7tM/CRJkiTtVhGJ4O3AURExnUoCeBrwttoDMnP6wPOI+DfgP4dL\nAiVJkiRJo9f0RDAzt0fE+4GVwATgqsy8MyLOru6/otkxSZIkSVKZFDJHMDNvBG4c0jZsApiZZzYj\nJkmSJEkqi0IWlJckSZIkFcdEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrG\nRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZE\nUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQ\nkiRJkkrpEnqvAAAgAElEQVSmkEQwIo6PiA0RsTEizh9m/8kRsS4i1kZET0S8oog4JUmSJKkd7dfs\nC0bEBOBy4I+AXuD2iLg+M++qOWwVcH1mZkTMBpYDL2p2rJIkSZLUjoroETwW2JiZ92bmk8Ay4OTa\nAzLz0czM6uaBQCJJkiRJGhNFJIJdwKaa7d5q2yARcUpE/BD4CvDuJsUmSZIkSW1v3BaLyczrMvNF\nwALgQ7s7LiLOqs4j7Nm8eXPzApQkSZKkFlVEItgHTK3ZnlJtG1Zm3gIcERGH7mb/0szszszuww47\nbGwjlSRJkqQ2VEQieDtwVERMj4j9gdOA62sPiIgjIyKqz18KPBN4qOmRSpIkSVIbanrV0MzcHhHv\nB1YCE4CrMvPOiDi7uv8K4E3AGRGxDegH3lpTPEaSJEmSVIdop/yqu7s7e3p6ig5jpxVr+liycgP3\nb+lncmcHi+bPYMHcXeriSJIklcK2bdvo7e3liSeeKDoUqeUdcMABTJkyhYkTJw5qj4jVmdm9t9c3\nvUewLFas6eOCa9fTv20HAH1b+rng2vUAJoOSJKmUent7Ofjgg5k2bRrVWUCS9kFm8tBDD9Hb28v0\n6dP36Rzjtmpoq1uycsPOJHBA/7YdLFm5oaCIJEmSivXEE0/wnOc8xyRQqlNE8JznPKeu3nUTwQa5\nf0v/qNolSZLKwCRQGhv1/lsyEWyQyZ0do2qXJElS4331q19lxowZHHnkkSxevHjYYzKTc889lyOP\nPJLZs2fzgx/8YFSvH+qggw4ak9j3xTe+8Q2+/e1vF3b9sTaSz3/JkiXMmTOHOXPmMHPmTCZMmMDD\nDz8MwLRp05g1axZz5syhu3uv0+iA9v3+TAQbZNH8GXRMnDCorWPiBBbNn1FQRJIkSeW2Y8cO3ve+\n93HTTTdx1113cc0113DXXXftctxNN93EPffcwz333MPSpUs555xzRvX6kdi+fXtd72Wk2ikRHOnn\nv2jRItauXcvatWu55JJLeNWrXsUhhxyyc//Xv/511q5dSz1FJtvh+zMRbJAFc7u45NRZdHV2EEBX\nZweXnDrLQjGSJEkjtGJNH/MW38z087/CvMU3s2JNX13n+/73v8+RRx7JEUccwf77789pp53Gl7/8\n5V2O+/KXv8wZZ5xBRPB7v/d7bNmyhQceeGDEr//JT37C7//+7zNr1iz+5m/+Zmf7N77xDV75yldy\n0kknccwxxwDwkY98hJkzZzJz5kw++tGPAnDffffxohe9iD/+4z/m6KOP5s1vfjOPP/44AKtWrWLu\n3LnMmjWLd7/73fzmN78BKj1dDz74IAA9PT28+tWv5r777uOKK67g0ksvZc6cOXzrW9+q6/MbtXXL\n4dKZcFFn5XHd8rpON9LPv9Y111zD6aefPqrrlOX7MxFsoAVzu7jt/Nfwk8Vv4LbzX2MSKEmSNEID\nFdj7tvSTPF2BvZ5ksK+vj6lTp+7cnjJlCn19u55vd8eN9PXnnXce55xzDuvXr+f5z3/+oH0/+MEP\n+NjHPsaPfvQjVq9ezac//Wm+973v8d3vfpdPfepTrFmzBoANGzbw3ve+l7vvvptnPetZfOITn+CJ\nJ57gzDPP5Itf/CLr169n+/btfPKTn9zt+502bRpnn302H/zgB1m7di2vfOUrR/5h1WvdcrjhXNi6\nCcjK4w3n1pUMjvTzH/D444/z1a9+lTe96U072yKC1772tbzsZS9j6dKlw76uLN+fiaAkSZLGnVau\nwH7bbbft7IV6xzveMWjfscceu7Pc/6233sopp5zCgQceyEEHHcSpp566s9dn6tSpzJs3D4C3v/3t\n3HrrrWzYsIHp06fzwhe+EIB3vvOd3HLLLc16W6Oz6mLYNqRI4rb+SnuT3HDDDcybN2/QsNBbb72V\ntWvXctNNN3H55ZcP+/mV5fszEWykMe4OlyRJKotGVGDv6upi06ZNO7d7e3vp6tp1xNbujhvp62H3\nFR0PPPDAEcU69PV7qxC533778dRTTwHUtaTAmNnaO7r2ERjN5w+wbNmyXYaFDhz/27/925xyyil8\n//vfH/a1Zfj+TAQbpQHd4ZIkSWXRiArsv/u7v8s999zDT37yE5588kmWLVvGSSedtMtxJ510Eldf\nfTWZyXe/+10mTZrE85///BG/ft68eSxbtgyAz3/+87uN55WvfCUrVqzg8ccf57HHHuO6667bOfzv\nZz/7Gd/5zncA+MIXvsArXvEKZsyYwX333cfGjRsB+OxnP8urXvUqoDKMcPXq1QB86Utf2nmNgw8+\nmEceeWRfPq76TJoyuvYRGOnnD7B161a++c1vcvLJJ+9se+yxx3Z+Fo899hhf+9rXmDlz5i6vLcv3\nZyLYKOOgO1ySJKlVNaIC+3777cfHP/5x5s+fz9FHH83ChQt58YtfDMAVV1zBFVdcAcDrX/96jjji\nCI488kj+9E//lE984hN7fX2tj33sY1x++eXMmjVrj3PYXvrSl3LmmWdy7LHH8vKXv5z3vOc9zJ07\nF4AZM2Zw+eWXc/TRR/OrX/2Kc845hwMOOIBPf/rTvOUtb2HWrFk84xnP4Oyzzwbg7/7u7zjvvPPo\n7u5mwoSnP7cTTzyR6667rvnFYo67ECYOSdondlTa99FIvz+A6667jte97nWDevB+8Ytf8IpXvIKX\nvOQlHHvssbzhDW/g+OOP3+U6Zfn+IjPH9IRF6u7uznrKwI6pizqB4T7bgIu2NDsaSZKkwt19990c\nffTRIz5+xZo+lqzcwP1b+pnc2cGi+TNKUXzvvvvu441vfCN33HFH0aHUZ93ySifI1t5KT+BxF8Ls\nhUVH1XDN/P6G+zcVEaszc6+LJO7XsKjKbtKU6rDQYdolSZK0VwvmdpUi8WtbsxeWIvFrVQ4NbZQG\ndIdLkiSp/U2bNq31ewNLrFW+PxPBRpm9EE68DCZNBaLyeOJl/l8RSZIkSYVzaGgj2R0uSZI0SGbu\ntZS+pL2rt9aLPYKSJElqigMOOICHHnqo7j9gpbLLTB566CEOOOCAfT6HPYKSJElqiilTptDb28vm\nzZuLDkVqeQcccABTpux7IUoTQUmSJDXFxIkTmT59etFhSMKhoZIkSZJUOiaCkiRJklQyJoKSJEmS\nVDLRTlWbImIz8NOi4xjGocCDRQehtuX9pUby/lIjeX+pkby/1Gjj9R77ncw8bG8HtVUiOF5FRE9m\ndhcdh9qT95cayftLjeT9pUby/lKjtfo95tBQSZIkSSoZE0FJkiRJKhkTweZYWnQAamveX2ok7y81\nkveXGsn7S43W0veYcwQlSZIkqWTsEZQkSZKkkjERbKCIOD4iNkTExog4v+h41F4i4r6IWB8RayOi\np+h41Poi4qqI+GVE3FHTdkhE/FdE3FN9fHaRMap17eb+uigi+qq/x9ZGxOuLjFGtKyKmRsTXI+Ku\niLgzIs6rtvs7THXbw/3V0r/DHBraIBExAfgR8EdAL3A7cHpm3lVoYGobEXEf0J2Z43H9GrWgiPhD\n4FHg6sycWW37MPBwZi6u/g+tZ2fmXxUZp1rTbu6vi4BHM/Ofi4xNrS8ing88PzN/EBEHA6uBBcCZ\n+DtMddrD/bWQFv4dZo9g4xwLbMzMezPzSWAZcHLBMUnSbmXmLcDDQ5pPBj5Tff4ZKv/hk0ZtN/eX\nNCYy84HM/EH1+SPA3UAX/g7TGNjD/dXSTAQbpwvYVLPdSxvcMBpXEvjviFgdEWcVHYza1nMz84Hq\n858Dzy0yGLWlD0TEuurQUYftqW4RMQ2YC3wPf4dpjA25v6CFf4eZCEqt6xWZOQc4AXhfddiV1DBZ\nmUvgfAKNpU8CRwBzgAeAfyk2HLW6iDgI+BLw55n569p9/g5TvYa5v1r6d5iJYOP0AVNrtqdU26Qx\nkZl91cdfAtdRGY4sjbVfVOdGDMyR+GXB8aiNZOYvMnNHZj4FfAp/j6kOETGRyh/pn8/Ma6vN/g7T\nmBju/mr132Emgo1zO3BUREyPiP2B04DrC45JbSIiDqxOViYiDgReB9yx51dJ++R64J3V5+8Evlxg\nLGozA3+gV52Cv8e0jyIigH8F7s7Mj9Ts8neY6ra7+6vVf4dZNbSBqiVkPwpMAK7KzH8oOCS1iYg4\ngkovIMB+wBe8v1SviLgGeDVwKPAL4O+AFcBy4HDgp8DCzLTgh0ZtN/fXq6kMqUrgPuDPauZzSSMW\nEa8AvgWsB56qNv81lXlc/g5TXfZwf51OC/8OMxGUJEmSpJJxaKgkSZIklYyJoCRJkiSVjImgJEmS\nJJWMiaAkSZIklYyJoCRJkiSVjImgJElDRMSOiFhb83P+GJ57WkS01FpTkqT2s1/RAUiSNA71Z+ac\nooOQJKlR7BGUJGmEIuK+iPhwRKyPiO9HxJHV9mkRcXNErIuIVRFxeLX9uRFxXUT8T/XnD6qnmhAR\nn4qIOyPiaxHRUdibkiSVkomgJEm76hgyNPStNfu2ZuYs4OPAR6tt/xf4TGbOBj4PXFZtvwz4Zma+\nBHgpcGe1/Sjg8sx8MbAFeFOD348kSYNEZhYdgyRJ40pEPJqZBw3Tfh/wmsy8NyImAj/PzOdExIPA\n8zNzW7X9gcw8NCI2A1My8zc155gG/FdmHlXd/itgYmb+fePfmSRJFfYISpI0Ormb56Pxm5rnO3DO\nviSpyUwEJUkanbfWPH6n+vzbwGnV538MfKv6fBVwDkBETIiISc0KUpKkPfH/QEqStKuOiFhbs/3V\nzBxYQuLZEbGOSq/e6dW2DwCfjohFwGbgXdX284ClEfEnVHr+zgEeaHj0kiTthXMEJUkaoeocwe7M\nfLDoWCRJqodDQyVJkiSpZOwRlCRJkqSSsUdQklQq1cXfMyKcJy9JKi0TQUlSS4mIr0bExcO0nxwR\nPzfBkyRp70wEJUmt5jPA2yMihrS/A/h8Zm4vIKYxERX+t1mS1HD+x0aS1GpWAM8BXjnQEBHPBt4I\nXF3dfkNErImIX0fEpoi4aKQnj4jzI+LHEfFIRNwVEacM2f+nEXF3zf6XVtunRsS1EbE5Ih6KiI9X\n2y+KiM/VvH7Q0NSI+EZE/ENE3AY8DhwREe+quca9EfFnQ2I4OSLWVt/fjyPi+Ih4S0SsHnLc/46I\nL4/0vUuSysNEUJLUUjKzH1gOnFHTvBD4YWb+T3X7ser+TuANwDkRsWCEl/gxlSRzEvD/AZ+LiOcD\nRMRbgIuq534WcBLwUERMAP4T+CkwDegClo3ibb0DOAs4uHqOX1JJbJ9FZU3CS2sSzmOpJLyLqu/v\nD4H7gOuB6RFx9JDzXj2KOCRJJWEiKElqRZ8B3hwRB1S3z6i2AZCZ38jM9Zn5VGauA64BXjWSE2fm\nv2fm/dXXfhG4Bzi2uvs9wIcz8/as2JiZP63unwwsyszHMvOJzLx1FO/n3zLzzszcnpnbMvMrmfnj\n6jW+CXyNp3tA/wS4KjP/qxpjX2b+MDN/A3wReDtARLyYSlL6n6OIQ5JUEiaCkqSWU02yHgQWRMQL\nqCRiXxjYHxEvj4ivV4dpbgXOBg4dybkj4ozqsMstEbEFmFnz2qlUegyHmgr8tI75iZuGxHBCRHw3\nIh6uxvD6EcQAlWT4bdX5k+8AllcTREmSBjERlCS1qqup9AS+HViZmb+o2fcFKkMlp2bmJOAKYGhx\nmV1ExO8AnwLeDzwnMzuBO2peuwl4wTAv3QQcvpuKpY8Bv1Wz/bxhjtm5qG9EPBP4EvDPwHOrMdw4\nghjIzO8CT1LpPXwb8NnhjpMkyURQktSqrgZeC/wpNcNCqw4GHs7MJ6pz6t42wnMeSCUp2wwQEe+i\n0iM44ErgLyLiZdUKn0dWk8fvAw8AiyPiwIg4ICLmVV+zFvjDiDg8IiYBF+wlhv2BZ1Zj2B4RJwCv\nq9n/r8C7IuK4iHhGRHRFxItq9l8NfBzYNsrhqZKkEjERlCS1pMy8D/g2leTt+iG73wtcHBGPABdS\nKS4zknPeBfwL8B3gF8As4Laa/f8O/AOVHsdHqFQwPSQzdwAnAkcCPwN6gbdWX/NfVOburQNWs5c5\ne5n5CHBuNeZfUUlir6/Z/32qBWSArcA3gd+pOcVnqSSvn0OSpN2IzNz7UZIkqSVERAeVqqMvzcx7\nio5HkjQ+2SMoSVJ7OQe43SRQkrQnw01qlyRJLSgi7qNSVGakayZKkkrKoaGSJEmSVDIODZUkSZKk\nkjERlCRJkqSSaas5goceemhOmzat6DAkSZIkqRCrV69+MDMP29txDU0EI+J44GPABODKzFw8ZP/J\nwIeAp4DtwJ/XLn4bEROAHqAvM9+4t+tNmzaNnp6eMXwHkiRJktQ6IuKnIzmuYUNDq0nc5cAJwDHA\n6RFxzJDDVgEvycw5wLuBK4fsPw+4u1ExSpIkSVIZNXKO4LHAxsy8NzOfBJYBJ9cekJmP5tNlSw8E\ndpYwjYgpwBvYNTmUJEmSJNWhkYlgF7CpZru32jZIRJwSET8EvkKlV3DAR4G/pDJsVJIkSZI0Rgqv\nGpqZ12Xmi6gsfvshgIh4I/DLzFy9t9dHxFkR0RMRPZs3b25wtJIkSZLU+hqZCPYBU2u2p1TbhpWZ\ntwBHRMShwDzgpIi4j8qQ0tdExOd287qlmdmdmd2HHbbX4jiSJEmSVHqNTARvB46KiOkRsT9wGnB9\n7QERcWRERPX5S4FnAg9l5gWZOSUzp1Vfd3Nmvr2BsUqSJElSaTRs+YjM3B4R7wdWUlk+4qrMvDMi\nzq7uvwJ4E3BGRGwD+oG31hSPkSRJ7Wjdclh1MWzthUlT4LgLYfbCoqOSpFKJdsq7uru703UEJUka\nx9YthxvOhW39T7dN7IATLzMZlKQxEBGrM7N7b8cVXixGkiSVyKqLByeBUNledXEx8UhSSZkISpKk\n5tnaO7p2SVJDmAhKkqTmmTRldO2SpIYwEZQkSc1z3IWVOYG1JnZU2iVJTWMiKEmSmmf2wkphmElT\ngag8WihGkpquYctHSJIkDWv2QhM/SSqYPYKSJEmSVDImgpIkSZJUMiaCkiRJklQyJoKSJEmSVDIm\ngpIkSZJUMiaCkiRJklQyJoKSJEmSVDImgpIkSZJUMiaCkiRJklQyJoKSJEmSVDImgpIkldW65XDp\nTLios/K4bnnREUmSmmS/ogOQJEkFWLccbjgXtvVXtrduqmwDzF5YXFySpKawR1CSpDJadfHTSeCA\nbf2VdklS2zMRlCSpjLb2jq5dktRWTAQlSSqjSVNG1y5JaisNTQQj4viI2BARGyPi/GH2nxwR6yJi\nbUT0RMQrqu1TI+LrEXFXRNwZEec1Mk5JkkrnuAthYsfgtokdlXZJUttrWCIYEROAy4ETgGOA0yPi\nmCGHrQJekplzgHcDV1bbtwP/JzOPAX4PeN8wr5UkSftq9kI48TKYNBWIyuOJl1kopt1YGbYc/J61\nDxpZNfRYYGNm3gsQEcuAk4G7Bg7IzEdrjj8QyGr7A8AD1eePRMTdQFftayVJUp1mLzTxa2dWhi0H\nv2fto0YODe0CNtVs91bbBomIUyLih8BXqPQKDt0/DZgLfK8hUUqSJLUjK8OWg9+z9lHhxWIy87rM\nfBGwAPhQ7b6IOAj4EvDnmfnr4V4fEWdV5xf2bN68ufEBS5Kk1lWmIXRWhi0Hv2fto0Ymgn3A1Jrt\nKdW2YWXmLcAREXEoQERMpJIEfj4zr93D65ZmZndmdh922GFjE7kkSWo/A0Potm4C8ukhdO2aDFoZ\nthz8nrWPGpkI3g4cFRHTI2J/4DTg+toDIuLIiIjq85cCzwQeqrb9K3B3Zn6kgTFKkqSyKNsQOivD\nloPfs/ZRw4rFZOb2iHg/sBKYAFyVmXdGxNnV/VcAbwLOiIhtQD/w1szM6jIS7wDWR8Ta6in/OjNv\nbFS8kiSpzZVtCN1AoZBVF1fe46QpleTAAiLtxe9Z+ygys+gYxkx3d3f29PQUHYYkSRqPLp1ZHRY6\nxKSp8ME7mh+PJDVARKzOzO69HVd4sRhJkqSmcAidJO1kIihJksph9kI48bJKDyBReTzxMofQSSql\nRi4oL0mSNL7MXmjiJ0nYIyhJktR4ZVq/UOXivd2y7BGUJElqpIH1CweWrhhYvxDsnVRr895uafYI\nSpIkNVLZ1i9UeXhvtzQTQUmSpEYq2/qFKg/v7ZZmIihJktRIk6aMrl1qFWW9t9tkXqSJoCRJUiO5\nfqHaVRnv7YF5kVs3Afn0vMgWTAZNBCVJkhrJ9QvLo016ikasjPd2G82LtGqoJElSo7l+YfsrawXN\nst3bbTQv0h5BSZIkqV5t1FOkPWijeZEmgpKk8aNsw6oktY826inSHrTRvEgTQUnS+NBGE/BHxeRX\nag9t1FOkPWijeZHOEZQkjQ97GlbVgv+BHZGyzimS2tFxFw7+9wwt21OkvWiTeZH2CEqSxocyDqty\nTpHUPtqop0jlYI+gJGlX65ZXkpGtvZVhTcdd2Pg/ZiZNqQ4LHaa9XZUx+ZUabMWaPpas3MD9W/qZ\n3NnBovkzWDC3qzkXb5OeIpWDPYKSpMGKmqvXRhPwR8w5RdKYWrGmjwuuXU/fln4S6NvSzwXXrmfF\nmr6iQ5PGHRNBSdJgRQ1XLHpYVRFFW8qY/EoNtGTlBvq37RjU1r9tB0tWbigoIjXKijV9zFt8M9PP\n/wrzFt9ssr8PHBoqSRqsyOGKRQ2rKqpoy8C5mz0MV2pT92/pH1W7WtNAz+9A0j/Q8ws0bxhwGzAR\nlCQNVsa5ekVWLHVOkTRmJnd20DdM0je5s2OYo9Wq9tTzayI4cg0dGhoRx0fEhojYGBHnD7P/5IhY\nFxFrI6InIl4x0tdKkhqkjMMVLdoitYVF82fQMXHCoLaOiRNYNH9GQRGpEez5HRsNSwQjYgJwOXAC\ncAxwekQcM+SwVcBLMnMO8G7gylG8VpLUCEXP1SuCRVuktrBgbheXnDqLrs4OAujq7OCSU2fZS9Rm\ndtfDa8/v6DRyaOixwMbMvBcgIpYBJwN3DRyQmY/WHH8gkCN9rSSpgco2XNGFoKW2sWBul4lfm1s0\nf8agOYJgz+++aOTQ0C6gdpJJb7VtkIg4JSJ+CHyFSq/giF8rSdKYKGMvqCS1KHt+x0bhxWIy8zrg\nuoj4Q+BDwGtH8/qIOAs4C+Dwww8f+wAlSeVQtl5QqV2tW24l3v+/vfuPkrOuDz3+/rgJZgVMENAD\n2aQJBSOahMSuYA0UFTUg8lONcBBFpRSqQj33pkKPjda2V1p6RbiCOZHCUYvG3AoBrkC0ICoimIRg\nItIAhQi7oEIkqcAiSfjcP2Y2TJbdZPbHs7Mzz/t1zp6Z5zvPM/OZmWdn57Pfz/f7LYET237CiS//\nPEzogpd3QNsiwPd5MIrsEewGptRsd1Tb+pWZPwIOiIh9BnNsZi7JzM7M7Nx3332HH7UkSZKaU+9S\nMJsfBfLFpWBGY11QjR7f5xFRZCK4EjgoIqZHxG7AKcD1tTtExIEREdXrbwReDmys51hpLHAxU0mS\nxpCdLQWj1uH7PCIKKw3NzK0R8QlgBdAGXJmZ90bE2dXbFwPvBT4UEVuAHuADmZlAv8cWFas0FC5m\nKknSGONSMOXg+zwiCh0jmJk3Ajf2aVtcc/2fgH+q91hpLHExU0mSxpiJHdVywX7a1Tp8n0dEoQvK\nS63MxUwlSRpjjlpUWfqllkvBtB7f5xHR8FlDpWa1/6R2uvtJ+lzMVCPK2e8kDcPyNd1ctGI9j23q\nYf9J7SycP6O1q1Z6Px/93Gxtvs8jIipD8lpDZ2dnrlq1qtFhqCT6jhGEymKmrmOjEdM7K1rfRc5d\n305SHfw7JZVTRKzOzM5d7WdpqDREDV/MdO0yuHgmfG5S5dIpk1uPs6JJGoadjWUfFf6dksY0S0Ol\nYThx7uTG/Fd17TK2XvdJxm17rrK9+dHKNthT1EpKOCta6crYpAI1dCx734qG3nXewL9T0hhhj6DU\nhJ69adGLSWDVuG3P8exNDpJuKQPNftais6L1lrF1b+oheXFJFtfnlIZmoDHrozKW3YoGacwzEZSa\n0ISeXw+qXU2qZLOiNbyMTWoxC+fPoH182w5t7ePbWDh/RvEPXsKKBqnZmAhKTeixF/YeVLua1OwF\nlYlhJk4BonLZwhPFuCSLNLIaOpa9ZBUNUjNyjKDUhK7Y7YP89ZbLeUU8v73t2dyNK3b7IJ9rXFgq\nwuwFLZv49eWSLNLIa9hY9qMW9T/rcYtWNEjNyB5BqQnNOfYsFuVZdL2wDy9k0PXCPizKs5hz7FnF\nP3gZZ4Er43NugIaWsUkaWSWraOi1fE038y68lennf5d5F946amOcG/W4am6uIyg1qYbMrljGde3K\n+JwbyFlDJTWrRq3b6HqR6qvedQRNBDVy1i6rzAa2uasyBuCoRX5RbjUXz6xMAd7XxCnwqV+Mfjyj\noYzPWZI0aPMuvLXf8vbJk9r5yflvb7nH1dhVbyLoGEGNjLKuF1S25LeMs8CV8TmXlL2Ro8fXWq2o\nURNeOdGWhsoxghoZZVwvqDf53fwokC8mv608fqyMs8CV8TmXkGsYjh5fa7WqRq3b2ND1ItXUTAQ1\nMsrYa1LG5Ldk69oB5XzOJeQahqPH17o8yjaBSaMmvHKiLQ2ViaBGRhl7TcqY/DZ6FrhGzN45ewEr\nZ/0dv2ZfXsjg1+zLyll/19olwCVkadXo8bUuhzL2/DZq3caGrheppuYYQY2MMq4XNLFjgElEWjj5\nhcata9egcajL13Rzwco/omfLJdvb2le28YUp3f6RbSGuYTh6fK3LYWc9v6382dmodRsbtl6kmpo9\nghoZje4pagRLBkdXg0pxLWMrB0urRo+vdTnY81seZSsBhtZ5zvYIauQ0qqeoUXqfa5lmDW2kBpXi\n+mWmHHr/k+5MlsXztS4He37Loe8ahr0lwEDL/k630nM2EZSGo2zJbyM1qBTXLzPlYWnV6PG1bn0L\n58/od5Fze35bSxlLgFvpOVsaKqk5NKgU1zI2Fa1VSoykWk5gUg5lrJpppedsj6Ck5tCgUlzL2FSk\nVr6CRiEAAB1kSURBVCoxkvqy57f1lbFqppWec6GJYEQcDVwCtAFXZOaFfW4/Dfg0EMDvgXMy8+fV\n2z4FnAkksA74SGY+V2S8ksa4BpXi+mVGRWmlEiNJ5VPGEuBWes6FlYZGRBtwGXAM8Hrg1Ih4fZ/d\nHgaOzMxZwN8DS6rHTgbOBTozcyaVRPKUomKVJKkRWqnESFL5lLEEuJWec5E9gocCD2bmQwARsRQ4\nAfhl7w6ZeUfN/ncCtbM+jAPaI2IL8ArgsQJjlaQxafmabstSW1grlRhJKqcyVs20ynMucrKYyUDt\nFH9d1baBfAy4CSAzu4F/AR4BHgc2Z+b3CopTksak3vFj3Zt6SF4cP+ZkIq3DyYgkSY0yJmYNjYi3\nUUkEP13d3otK7+F0YH9g94j44ADHnhURqyJi1RNPPDFaIUtS4VzMvvW1UomRds7ZYSWNNUWWhnYD\nU2q2O6ptO4iI2cAVwDGZubHa/A7g4cx8orrPNcBbgH/re3xmLqE6trCzszNH8glIUiM5fqwcWqXE\nSANzdlhJY1GRPYIrgYMiYnpE7EZlspfra3eIiKnANcDpmXl/zU2PAG+OiFdERABHAfcVGKskjTkD\njRNz/JjUXOzdlzQW7TIRjIhPVks1ByUztwKfAFZQSeKWZea9EXF2RJxd3W0RsDdweUTcExGrqsfe\nBfw7cDeVpSNeRrXXT5LKwvFjUmuwd1/SWFRPaehrgJURcTdwJbAiM+sqwczMG4Eb+7Qtrrl+JpW1\nAvs79rPAZ+t5HElqRS5mL7UGZ4eVNBZFPTldtTzzXcBHgE5gGfCvmflfxYY3OJ2dnblq1apGhyFJ\nkrRd3zGCUOndd2IgSUWIiNWZ2bmr/eoaI1jtAfx19WcrsBfw7xHxz8OKUpIkqcU5O6yksWiXPYIR\ncR7wIeBJKrN7Ls/MLRHxMuCBzPzj4sOsjz2CkiRJksqs3h7BesYIvgo4OTN/VduYmS9ExHuGGqAk\nSZIkqTHqKQ29Cfhd70ZEvDIiDgPITJd00NiwdhlcPBM+N6lyuXZZoyOSJEmSxqx6EsGvAE/XbD9d\nbZPGhrXL4IZzYfOjQFYubzjXZFCSJEkaQD2JYNQuF5GZL1BfSak0Om75PGzpMy33lp5KuyRJkqSX\nqCcRfCgizo2I8dWf84CHig5MqtvmrsG1S5IkSSVXT8/e2cClwGeABG4BzioyKDWn5Wu6G7Pw9cSO\nalloP+2ShqRhv8+SJGlU7DIRzMzfAqeMQixqYn0Xy+3e1MMF16wDKP7L41GLKmMCa8tDx7dX2iUN\nWkN/nyVJ0qjYZWloREyIiI9HxOURcWXvz2gEp+Zx0Yr127809urZso2LVqwv/sFnL4DjLoWJU4Co\nXB53aaVd0qA19PdZkiSNinpKQ78B/CcwH/g8cBrgshHawWObegbVPuJmLzDxk0ZIw3+fNWosAZak\n8qpnspgDM/NvgWcy82vAscBhxYalZrP/pPZBtUsau/x9LofeEuDuTT0kL5YAL1/T3ejQJEmjoJ5E\ncEv1clNEzAQmAq8uLiQ1o4XzZ9A+vm2HtvbxbSycP6NBEakoy9d0M+/CW5l+/neZd+GtfmlsQf4+\nl4MlwJJUbvWUhi6JiL2ozBp6PbAH8LeFRqWm01tKZIlRa3MSkXLw97kcyloCbDmsJFXsNBGMiJcB\n/52ZTwE/Ag4YlajUlE6cO9k/pi1uZz0Ivvetxd/n1rf/pHa6+0n6WrkE2H9mSdKLdloampkvAH89\nSrFIGuPK2oMgtaIylgBbDitJL6qnNPQ/IuJ/At8GnultzMzfFRaVhsWyFxWljD0IUqsqYwmw/8yS\npBfVkwh+oHr58Zq2xDLRMcmyFxVp4fwZO5xf0Po9CFIrK1sJsP/MkqQX7XLW0Myc3s+PSeAYZdmL\ninTi3Ml84eRZTJ7UTgCTJ7XzhZNnleqLpKTmVcZyWEkayC57BCPiQ/21Z+bXRz4cDZdlLypa2XoQ\nJLWOMpbDStJA6ikNfVPN9QnAUcDdwC4TwYg4GrgEaAOuyMwL+9x+GvBpIIDfA+dk5s+rt00CrgBm\nUilF/Whm/rSOeEvNshdJkgbmP7MkqWKXiWBmfrJ2u5qgLd3VcRHRBlwGvBPoAlZGxPWZ+cua3R4G\njszMpyLiGGAJcFj1tkuAmzPzfRGxG/CKep5Q2TmGS5IkSdKu1NMj2NczwPQ69jsUeDAzHwKIiKXA\nCcD2RDAz76jZ/06go7rvRODPgDOq+z0PPD+EWEvHshdJkiRJu1LPGMEbqJRmQmVymdcDy+q478nA\nozXbXbzY29efjwE3Va9PB54AroqIQ4DVwHmZ+cxAB+tFlr1IkiRJ2pl6egT/peb6VuBXmdk1kkFE\nxNuoJIKH18T1RuCTmXlXRFwCnA/8bT/HngWcBTB16tSRDEuSJEmSWtIul48AHgHuyswfZuZPgI0R\nMa2O47qBKTXbHdW2HUTEbCqTwpyQmRurzV1AV2beVd3+dyqJ4Utk5pLM7MzMzn333beOsCRJkiSp\n3OpJBP8v8ELN9rZq266sBA6KiOnVyV5OAa6v3SEipgLXAKdn5v297Zn5a+DRiOid4eQoasYWSpIk\nSZKGrp7S0HHVyVqAysQt1cRupzJza0R8AlhBZfmIKzPz3og4u3r7YmARsDdweUQAbM3MzupdfBK4\nuvpYDwEfGcTzKre1y+CWz8PmLpjYAUctgtkLGh2VJEmSpDGinkTwiYg4PjOvB4iIE4An67nzzLwR\nuLFP2+Ka62cCZw5w7D1AZ3+3aSfWLoMbzoUt1bUENz9a2QaTQUmSJElAfaWhZwN/ExGPRMQjVBaA\n/4tiw9KQ3fL5F5PAXlt6Ku2SJEmSRH0Lyv8X8OaI2KO6/XThUWnoNg8woetA7ZIkSZJKZ5c9ghHx\nvyJiUmY+nZlPR8ReEfEPoxGchmBix+DaJUmSJJVOPaWhx2Tmpt6NzHwKeHdxIWlYjloE49t3bBvf\nXmmXJEmSJOpLBNsi4uW9GxHRDrx8J/urkWYvgOMuhYlTgKhcHnepE8VIkiRJ2q6eWUOvBm6JiKuA\nAM4AvlZkUBqm2QtM/CRJkiQNqJ7JYv4pIn4OvANIKusC/lHRgUmSJEmSilFPaSjAb6gkge8H3g7c\nV1hEkiRJkqRCDdgjGBGvBU6t/jwJfBuIzHzbKMUmSZIkSSrAzkpD/xP4MfCezHwQICI+NSpRSZIk\nSZIKs7NE8GTgFOAHEXEzsJTKZDHSmLN8TTcXrVjPY5t62H9SOwvnz+DEuZMbHZYkSZI0Jg04RjAz\nl2fmKcDrgB8AfwW8OiK+EhHvGq0ApV1ZvqabC65ZR/emHhLo3tTDBdesY/ma7kaHJkmSJI1Ju5ws\nJjOfycxvZuZxQAewBvh04ZFJdbpoxXp6tmzboa1nyzYuWrG+QRFJkiRJY1u9s4YCkJlPZeaSzDyq\nqICkwXpsU8+g2iVJkqSyG1QiKI1F+09qH1S7JEmSVHYmgmp6C+fPoH182w5t7ePbWDh/RoMikiRJ\nksa2nc0aKjWF3tlBnTVUkiRJqo+JoFrCiXMnm/hJkiRJdbI0VJIkSZJKxkRQkiRJkkrGRFCSJEmS\nSqbQRDAijo6I9RHxYESc38/tp0XE2ohYFxF3RMQhfW5vi4g1EfH/ioxTkiRJksqksMliIqINuAx4\nJ9AFrIyI6zPzlzW7PQwcmZlPRcQxwBLgsJrbzwPuA15ZVJySmsfyNd3ODitJkjQCiuwRPBR4MDMf\nyszngaXACbU7ZOYdmflUdfNOoKP3tojoAI4FrigwRklNYvmabi64Zh3dm3pIoHtTDxdcs47la7ob\nHZokSVLTKTIRnAw8WrPdVW0byMeAm2q2vwT8NfDCyIcmqdlctGI9PVu27dDWs2UbF61Y36CIJEmS\nmteYmCwmIt5GJRH8dHX7PcBvM3N1HceeFRGrImLVE088UXCkkhrlsU09g2qXJEnSwIpMBLuBKTXb\nHdW2HUTEbCrlnydk5sZq8zzg+IjYQKWk9O0R8W/9PUhmLsnMzszs3HfffUcyfkljyP6T2gfVLkmS\npIEVmQiuBA6KiOkRsRtwCnB97Q4RMRW4Bjg9M+/vbc/MCzKzIzOnVY+7NTM/WGCsksa4hfNn0D6+\nbYe29vFtLJw/o0ERSZIkNa/CZg3NzK0R8QlgBdAGXJmZ90bE2dXbFwOLgL2ByyMCYGtmdhYVk6Tm\n1Ts7qLOGSpIkDV9kZqNjGDGdnZ25atWqRochSZIkSQ0REavr6VwbE5PFSJIkSZJGj4mgJEmSJJWM\niaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSZIklYyJ\noCRJkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJkiSVjImg\nJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJkiSVTKGJYEQcHRHrI+LB\niDi/n9tPi4i1EbEuIu6IiEOq7VMi4gcR8cuIuDcizisyTkmSJEkqk3FF3XFEtAGXAe8EuoCVEXF9\nZv6yZreHgSMz86mIOAZYAhwGbAX+R2beHRF7Aqsj4vt9jpUkSZIkDUGRPYKHAg9m5kOZ+TywFDih\ndofMvCMzn6pu3gl0VNsfz8y7q9d/D9wHTC4wVkmSJEkqjSITwcnAozXbXew8mfsYcFPfxoiYBswF\n7hrB2CRJkiSptAorDR2MiHgblUTw8D7tewDfAf4qM/97gGPPAs4CmDp1asGRSpIkSVLzK7JHsBuY\nUrPdUW3bQUTMBq4ATsjMjTXt46kkgVdn5jUDPUhmLsnMzszs3HfffUcseEmSJElqVUUmgiuBgyJi\nekTsBpwCXF+7Q0RMBa4BTs/M+2vaA/hX4L7M/GKBMUqSJElS6RRWGpqZWyPiE8AKoA24MjPvjYiz\nq7cvBhYBewOXV3I/tmZmJzAPOB1YFxH3VO/ybzLzxqLilSRJkqSyiMxsdAwjprOzM1etWtXoMCRJ\nkiSpISJidbVzbacKXVBekiRJkjT2mAhKkiRJUsmYCEqSJElSyZgISpIkSVLJmAhKkiRJUsmYCEqS\nJElSyZgISpIkSVLJmAhKkiRJUsmYCEqSJElSyZgISpIkSVLJmAhKkiRJUsmYCEqSJElSyZgISpIk\nSVLJmAhKkiRJUsmYCEqSJElSyZgISpIkSVLJmAhKkiRJUsmYCEqSJElSyZgISpIkSVLJmAhKkiRJ\nUsmYCEqSJElSyRSaCEbE0RGxPiIejIjz+7n9tIhYGxHrIuKOiDik3mMlSZIkSUNTWCIYEW3AZcAx\nwOuBUyPi9X12exg4MjNnAX8PLBnEsZIkSZKkISiyR/BQ4MHMfCgznweWAifU7pCZd2TmU9XNO4GO\neo9tCmuXwcUz4XOTKpdrlzU6IkmSJEkqNBGcDDxas91VbRvIx4Cbhnjs2LN2GdxwLmx+FMjK5Q3n\nmgxKkiRJarhxjQ4AICLeRiURPHwIx54FnAUwderUEY5sGG75PGzp2bFtS0+lffaCxsQkSZLUQFu2\nbKGrq4vnnnuu0aFITW/ChAl0dHQwfvz4IR1fZCLYDUyp2e6otu0gImYDVwDHZObGwRwLkJlLqI4t\n7OzszOGHPUI2dw2uXZIkqcV1dXWx5557Mm3aNCKi0eFITSsz2bhxI11dXUyfPn1I91FkaehK4KCI\nmB4RuwGnANfX7hARU4FrgNMz8/7BHDvmTewYXLskSVKLe+6559h7771NAqVhigj23nvvYfWuF5YI\nZuZW4BPACuA+YFlm3hsRZ0fE2dXdFgF7A5dHxD0RsWpnxxYVayGOWgTj23dsG99eaZckSSopk0Bp\nZAz3d6nQdQQz88bMfG1m/nFm/mO1bXFmLq5ePzMz98rMOdWfzp0d21RmL4DjLoWJU4CoXB53qeMD\nJUmSGujmm29mxowZHHjggVx44YX97pOZnHvuuRx44IHMnj2bu+++e1DH97XHHnuMSOxDcdttt3HH\nHXc07PFHWj2v/0UXXcScOXOYM2cOM2fOpK2tjd/97ncATJs2jVmzZjFnzhw6Ozv7Pb6vVn3/xsRk\nMS1r9gITP0mSpCFavqabi1as57FNPew/qZ2F82dw4tyhTyS/bds2Pv7xj/P973+fjo4O3vSmN3H8\n8cfz+tfvuFz1TTfdxAMPPMADDzzAXXfdxTnnnMNdd91V9/H12Lp1K+PGFf9V/LbbbmOPPfbgLW95\nS+GP9RJrl1UmStzcVRkeddSiYX03rvf1X7hwIQsXLgTghhtu4OKLL+ZVr3rV9tt/8IMfsM8++ww5\nDmiN96/QHkFJkiRpKJav6eaCa9bRvamHBLo39XDBNetYvqbf+QPr8rOf/YwDDzyQAw44gN12241T\nTjmF66677iX7XXfddXzoQx8iInjzm9/Mpk2bePzxx+s+/uGHH+ZP//RPmTVrFp/5zGe2t992220c\nccQROyQvX/ziF5k5cyYzZ87kS1/6EgAbNmzgda97HaeddhoHH3ww73vf+3j22WcBuOWWW5g7dy6z\nZs3iox/9KH/4wx+ASk/Xk08+CcCqVat461vfyoYNG1i8eDEXX3wxc+bM4cc//vGQX7tBK2AptXpf\n/1rf+ta3OPXUUwf1OGV5/0wEJUmSNOZctGI9PVu27dDWs2UbF61YP+T77O7uZsqUFyem7+jooLv7\npYnlQPvVe/x5553HOeecw7p169hvv/12uO3uu+/mkksu4f7772f16tVcddVV3HXXXdx555189atf\nZc2aNQCsX7+ev/zLv+S+++7jla98JZdffjnPPfccZ5xxBt/+9rdZt24dW7du5Stf+cqAz3fatGmc\nffbZfOpTn+Kee+7hiCOOqP/FGq6dLaU2RPW+/r2effZZbr75Zt773vdub4sI3vGOd/Anf/InLFmy\npN/jyvL+mQhKkiRpzHlsU8+g2seSn/zkJ9t7oU4//fQdbjv00EO3T/d/++23c9JJJ7H77ruzxx57\ncPLJJ2/v9ZkyZQrz5s0D4IMf/CC3334769evZ/r06bz2ta8F4MMf/jA/+tGPRutpDc4YWErthhtu\nYN68eTuUhd5+++3cc8893HTTTVx22WX9vn5lef9MBAu0fE038y68lennf5d5F946rFIGSZKkMtl/\nUvug2usxefJkHn300e3bXV1dTJ780jGHA+1X7/Ew8IyOu+++e12x9j1+VzNEjhs3jhdeeAFgWEsK\njJgCllIbzOsPsHTp0peUhfbu/+pXv5qTTjqJn/3sZ/0eW4b3z0SwIEXUtUuSJJXFwvkzaB/ftkNb\n+/g2Fs6fMeT7fNOb3sQDDzzAww8/zPPPP8/SpUs5/vjjX7Lf8ccfz9e//nUykzvvvJOJEyey3377\n1X38vHnzWLp0KQBXX331gPEcccQRLF++nGeffZZnnnmGa6+9dnv53yOPPMJPf/pTAL75zW9y+OGH\nM2PGDDZs2MCDDz4IwDe+8Q2OPPJIoFJGuHr1agC+853vbH+MPffck9///vdDebmGp4Cl1Op9/QE2\nb97MD3/4Q0444YTtbc8888z21+KZZ57he9/7HjNnznzJsWV5/0wEC1JEXbskSVJZnDh3Ml84eRaT\nJ7UTwORJ7Xzh5FnDmjV03LhxfPnLX2b+/PkcfPDBLFiwgDe84Q0ALF68mMWLFwPw7ne/mwMOOIAD\nDzyQP//zP+fyyy/f5fG1LrnkEi677DJmzZq10zFsb3zjGznjjDM49NBDOeywwzjzzDOZO3cuADNm\nzOCyyy7j4IMP5qmnnuKcc85hwoQJXHXVVbz//e9n1qxZvOxlL+PssyvLc3/2s5/lvPPOo7Ozk7a2\nFxPo4447jmuvvXb0J4spYCm1et8/gGuvvZZ3vetdO/Tg/eY3v+Hwww/nkEMO4dBDD+XYY4/l6KOP\nfsnjlOX9i8wc0TtspM7Ozly1alWjwwBg+vnfpb9XNoCHLzx2tMORJElquPvuu4+DDz640WGMeRs2\nbOA973kPv/jFLxodioZgNN+//n6nImJ17frsA7FHsCBF1LVLkiRJ0kgwESxIEXXtkiRJan3Tpk2z\nN7CJNcv7N67RAbSq3vr1i1as57FNPew/qZ2F82cMq65dkiRJkkaCiWCBTpw72cRPkiSpRmbucip9\nSbs23LleLA2VJEnSqJgwYQIbN24c9hdYqewyk40bNzJhwoQh34c9gpIkSRoVHR0ddHV18cQTTzQ6\nFKnpTZgwgY6OjiEfbyIoSZKkUTF+/HimT5/e6DAkYWmoJEmSJJWOiaAkSZIklYyJoCRJkiSVTLTS\nrE0R8QTwq0bH0Y99gCcbHYRalueXiuT5pSJ5fqlInl8q2lg9x/4oM/fd1U4tlQiOVRGxKjM7Gx2H\nWpPnl4rk+aUieX6pSJ5fKlqzn2OWhkqSJElSyZgISpIkSVLJmAiOjiWNDkAtzfNLRfL8UpE8v1Qk\nzy8VranPMccISpIkSVLJ2CMoSZIkSSVjIligiDg6ItZHxIMRcX6j41FriYgNEbEuIu6JiFWNjkfN\nLyKujIjfRsQvatpeFRHfj4gHqpd7NTJGNa8Bzq/PRUR39XPsnoh4dyNjVPOKiCkR8YOI+GVE3BsR\n51Xb/QzTsO3k/GrqzzBLQwsSEW3A/cA7gS5gJXBqZv6yoYGpZUTEBqAzM8fi+jVqQhHxZ8DTwNcz\nc2a17Z+B32XmhdV/aO2VmZ9uZJxqTgOcX58Dns7Mf2lkbGp+EbEfsF9m3h0RewKrgROBM/AzTMO0\nk/NrAU38GWaPYHEOBR7MzIcy83lgKXBCg2OSpAFl5o+A3/VpPgH4WvX616j84ZMGbYDzSxoRmfl4\nZt5dvf574D5gMn6GaQTs5PxqaiaCxZkMPFqz3UULnDAaUxL4j4hYHRFnNToYtazXZObj1eu/Bl7T\nyGDUkj4ZEWurpaOW7WnYImIaMBe4Cz/DNML6nF/QxJ9hJoJS8zo8M+cAxwAfr5ZdSYXJylgCxxNo\nJH0FOACYAzwO/O/GhqNmFxF7AN8B/ioz/7v2Nj/DNFz9nF9N/RlmIlicbmBKzXZHtU0aEZnZXb38\nLXAtlXJkaaT9pjo2oneMxG8bHI9aSGb+JjO3ZeYLwFfxc0zDEBHjqXxJvzozr6k2+xmmEdHf+dXs\nn2EmgsVZCRwUEdMjYjfgFOD6BsekFhERu1cHKxMRuwPvAn6x86OkIbke+HD1+oeB6xoYi1pM7xf0\nqpPwc0xDFBEB/CtwX2Z+seYmP8M0bAOdX83+GeasoQWqTiH7JaANuDIz/7HBIalFRMQBVHoBAcYB\n3/T80nBFxLeAtwL7AL8BPgssB5YBU4FfAQsy0wk/NGgDnF9vpVJSlcAG4C9qxnNJdYuIw4EfA+uA\nF6rNf0NlHJefYRqWnZxfp9LEn2EmgpIkSZJUMpaGSpIkSVLJmAhKkiRJUsmYCEqSJElSyZgISpIk\nSVLJmAhKkiRJUsmYCEqS1EdEbIuIe2p+zh/B+54WEU211pQkqfWMa3QAkiSNQT2ZOafRQUiSVBR7\nBCVJqlNEbIiIf46IdRHxs4g4sNo+LSJujYi1EXFLREyttr8mIq6NiJ9Xf95Svau2iPhqRNwbEd+L\niPaGPSlJUimZCEqS9FLtfUpDP1Bz2+bMnAV8GfhSte3/AF/LzNnA1cCl1fZLgR9m5iHAG4F7q+0H\nAZdl5huATcB7C34+kiTtIDKz0TFIkjSmRMTTmblHP+0bgLdn5kMRMR74dWbuHRFPAvtl5pZq++OZ\nuU9EPAF0ZOYfau5jGvD9zDyouv1pYHxm/kPxz0ySpAp7BCVJGpwc4Ppg/KHm+jYcsy9JGmUmgpIk\nDc4Hai5/Wr1+B3BK9fppwI+r128BzgGIiLaImDhaQUqStDP+B1KSpJdqj4h7arZvzszeJST2ioi1\nVHr1Tq22fRK4KiIWAk8AH6m2nwcsiYiPUen5Owd4vPDoJUnaBccISpJUp+oYwc7MfLLRsUiSNByW\nhkqSJElSydgjKEmSJEklY4+gJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSZIklYyJ\noCRJkiSVzP8Hr1ZUS8bEGPcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x16132766f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question\n",
    "Explain what you see in this experiment. What does it suggest about dropout?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Answer\n",
    "We get slightly worse at training time to get better results at test time. Dropout is acting as a regularizer. It prevents models overfitting to training data and as a result making them generalise better at test time."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
